Skip to content

feat(cluster): automatic failover via epoch-based voting - #189

Merged
kacy merged 2 commits into
mainfrom
feat/auto-failover
Feb 19, 2026
Merged

feat(cluster): automatic failover via epoch-based voting#189
kacy merged 2 commits into
mainfrom
feat/auto-failover

Conversation

@kacy

@kacy kacy commented Feb 19, 2026

Copy link
Copy Markdown
Owner

Summary

implements automatic failover for primary nodes that gossip confirms as dead.
replicas of the failed node start an epoch-based election, collect votes from
a majority of the remaining primaries, and promote themselves without manual
intervention.

  • crates/ember-cluster/src/election.rs (new) — election state machine
    tracking epoch, votes received, and quorum (n/2 + 1)
  • crates/ember-cluster/src/message.rs — two new NodeUpdate variants:
    VoteRequest { candidate, epoch, offset } (tag 7) and
    VoteGranted { from, candidate, epoch } (tag 8), both serialised in the
    existing binary gossip wire format
  • crates/ember-cluster/src/gossip.rs — matching GossipEvent variants;
    apply_updates() emits them; queue_vote_request / queue_vote_granted
    helpers on GossipEngine
  • crates/ember-server/src/cluster.rs — three new methods on
    ClusterCoordinator:
    • start_election(failed_primary) — stagger delay + broadcast vote request
    • handle_vote_request(candidate, epoch) — primaries grant one vote per epoch
    • handle_vote_granted(from, candidate, epoch) — candidates track quorum,
      trigger cluster_failover(force=true) when reached

the gossip event consumer detects MemberFailed for our primary and spawns an
election task. vote events are handled via a PostAction enum so the state
write-lock is released before calling gossip or async failover methods.

What was tested

  • all 95 ember-cluster unit tests pass (gossip, message roundtrips,
    election state machine, topology)
  • all 72 ember-server unit tests pass, including four new election tests:
    • primary_grants_vote_once_per_epoch
    • replica_does_not_grant_vote
    • vote_granted_reaches_quorum_and_promotes
    • vote_granted_wrong_candidate_ignored
  • cargo clippy --workspace -- -D warnings clean

Design considerations

stagger delay is a fixed 500 ms today. in production, the delay should be
proportional to (max_offset - my_offset) so the most up-to-date replica wins.
this requires either a gossip-advertised max offset or a coordination round;
left as a follow-up since the correctness of the voting algorithm does not
depend on it.

vote scope is intentionally limited to primaries. replicas that are not
candidates never vote, preventing split-brain from out-of-date nodes.

epoch enforcement (last_voted_epoch atomic) ensures each primary votes
at most once per config epoch, regardless of how many VoteRequest messages
it receives from competing candidates.

kacy added 2 commits February 18, 2026 20:56
adds two new NodeUpdate variants — VoteRequest and VoteGranted — that
are piggybacked on Ping/Ack messages to coordinate automatic failover
elections. also adds the matching GossipEvent variants that the server
layer consumes, and queue_vote_request / queue_vote_granted helpers on
GossipEngine.

the new election.rs module tracks election state: epoch, votes received,
and quorum calculation. a majority (n/2 + 1) of alive primaries must
vote for a candidate before it can promote.
when gossip confirms a primary as dead and this node is one of its
replicas, we start an election:
- wait 500ms (stagger delay lets more up-to-date replicas win)
- broadcast VoteRequest via gossip piggybacking
- primaries that haven't voted in this epoch respond with VoteGranted
- first replica to reach majority quorum promotes itself via FAILOVER FORCE

VoteRequest events are handled by primaries (one vote per epoch, enforced
by last_voted_epoch atomic). VoteGranted events are handled by the
candidate; quorum triggers cluster_failover(force=true).

post-lock actions (StartElection, HandleVoteRequest, HandleVoteGranted)
are dispatched from the gossip event consumer after releasing the state
write-lock to avoid deadlocks.
@kacy
kacy merged commit 6df0193 into main Feb 19, 2026
4 of 7 checks passed
@kacy
kacy deleted the feat/auto-failover branch February 19, 2026 01:57
kacy added a commit that referenced this pull request Feb 19, 2026
* feat(cluster): add VoteRequest/VoteGranted gossip updates for elections

adds two new NodeUpdate variants — VoteRequest and VoteGranted — that
are piggybacked on Ping/Ack messages to coordinate automatic failover
elections. also adds the matching GossipEvent variants that the server
layer consumes, and queue_vote_request / queue_vote_granted helpers on
GossipEngine.

the new election.rs module tracks election state: epoch, votes received,
and quorum calculation. a majority (n/2 + 1) of alive primaries must
vote for a candidate before it can promote.

* feat(server): automatic failover via epoch-based voting

when gossip confirms a primary as dead and this node is one of its
replicas, we start an election:
- wait 500ms (stagger delay lets more up-to-date replicas win)
- broadcast VoteRequest via gossip piggybacking
- primaries that haven't voted in this epoch respond with VoteGranted
- first replica to reach majority quorum promotes itself via FAILOVER FORCE

VoteRequest events are handled by primaries (one vote per epoch, enforced
by last_voted_epoch atomic). VoteGranted events are handled by the
candidate; quorum triggers cluster_failover(force=true).

post-lock actions (StartElection, HandleVoteRequest, HandleVoteGranted)
are dispatched from the gossip event consumer after releasing the state
write-lock to avoid deadlocks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant